{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run the following from the cs231n directory and try again:\n",
      "python setup.py build_ext --inplace\n",
      "You may also need to restart your iPython kernel\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?\n",
    "\n",
    "## Answer:\n",
    "如果训练中不对 dropout 层进行缩放，那么测试时由于神经元数量增加，dropout 层的输入会被放大\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the network receives a value that is not 1 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302371489704412\n",
      "W1 relative error: 1.90e-07\n",
      "W2 relative error: 4.76e-06\n",
      "W3 relative error: 2.60e-08\n",
      "b1 relative error: 4.73e-09\n",
      "b2 relative error: 1.82e-09\n",
      "b3 relative error: 1.70e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3042759220785896\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 1.84e-08\n",
      "W3 relative error: 5.35e-08\n",
      "b1 relative error: 2.58e-08\n",
      "b2 relative error: 2.99e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856644\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.878000; val_acc: 0.269000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.275000\n",
      "(Epoch 10 / 25) train acc: 0.888000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.926000; val_acc: 0.277000\n",
      "(Epoch 12 / 25) train acc: 0.960000; val_acc: 0.302000\n",
      "(Epoch 13 / 25) train acc: 0.964000; val_acc: 0.307000\n",
      "(Epoch 14 / 25) train acc: 0.966000; val_acc: 0.309000\n",
      "(Epoch 15 / 25) train acc: 0.976000; val_acc: 0.289000\n",
      "(Epoch 16 / 25) train acc: 0.988000; val_acc: 0.303000\n",
      "(Epoch 17 / 25) train acc: 0.988000; val_acc: 0.309000\n",
      "(Epoch 18 / 25) train acc: 0.994000; val_acc: 0.312000\n",
      "(Epoch 19 / 25) train acc: 0.992000; val_acc: 0.317000\n",
      "(Epoch 20 / 25) train acc: 0.988000; val_acc: 0.317000\n",
      "(Iteration 101 / 125) loss: 0.139997\n",
      "(Epoch 21 / 25) train acc: 0.992000; val_acc: 0.295000\n",
      "(Epoch 22 / 25) train acc: 0.982000; val_acc: 0.295000\n",
      "(Epoch 23 / 25) train acc: 0.986000; val_acc: 0.286000\n",
      "(Epoch 24 / 25) train acc: 0.994000; val_acc: 0.305000\n",
      "(Epoch 25 / 25) train acc: 0.994000; val_acc: 0.295000\n",
      "\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318479\n",
      "(Epoch 0 / 25) train acc: 0.230000; val_acc: 0.177000\n",
      "(Epoch 1 / 25) train acc: 0.378000; val_acc: 0.243000\n",
      "(Epoch 2 / 25) train acc: 0.402000; val_acc: 0.254000\n",
      "(Epoch 3 / 25) train acc: 0.502000; val_acc: 0.276000\n",
      "(Epoch 4 / 25) train acc: 0.528000; val_acc: 0.298000\n",
      "(Epoch 5 / 25) train acc: 0.562000; val_acc: 0.297000\n",
      "(Epoch 6 / 25) train acc: 0.628000; val_acc: 0.291000\n",
      "(Epoch 7 / 25) train acc: 0.622000; val_acc: 0.299000\n",
      "(Epoch 8 / 25) train acc: 0.684000; val_acc: 0.311000\n",
      "(Epoch 9 / 25) train acc: 0.714000; val_acc: 0.291000\n",
      "(Epoch 10 / 25) train acc: 0.724000; val_acc: 0.297000\n",
      "(Epoch 11 / 25) train acc: 0.762000; val_acc: 0.303000\n",
      "(Epoch 12 / 25) train acc: 0.786000; val_acc: 0.284000\n",
      "(Epoch 13 / 25) train acc: 0.824000; val_acc: 0.303000\n",
      "(Epoch 14 / 25) train acc: 0.826000; val_acc: 0.347000\n",
      "(Epoch 15 / 25) train acc: 0.862000; val_acc: 0.349000\n",
      "(Epoch 16 / 25) train acc: 0.862000; val_acc: 0.315000\n",
      "(Epoch 17 / 25) train acc: 0.870000; val_acc: 0.307000\n",
      "(Epoch 18 / 25) train acc: 0.876000; val_acc: 0.340000\n",
      "(Epoch 19 / 25) train acc: 0.886000; val_acc: 0.327000\n",
      "(Epoch 20 / 25) train acc: 0.880000; val_acc: 0.322000\n",
      "(Iteration 101 / 125) loss: 3.870114\n",
      "(Epoch 21 / 25) train acc: 0.894000; val_acc: 0.325000\n",
      "(Epoch 22 / 25) train acc: 0.898000; val_acc: 0.306000\n",
      "(Epoch 23 / 25) train acc: 0.878000; val_acc: 0.306000\n",
      "(Epoch 24 / 25) train acc: 0.914000; val_acc: 0.321000\n",
      "(Epoch 25 / 25) train acc: 0.910000; val_acc: 0.328000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver\n",
    "  print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4gAAAJNCAYAAABz4lt9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XucnGV5+P/PZVhkRcxyiEiygQRJgzQJxC4gQgFJIagcAmoED4gnxErxZ9tU6K/NN+ZbCyVULIXWRkXAA5BijMECUfGIB2RjcCNgJCKS3WANh0TUBUO4vn/MbJgsu0k2mZlndvbzfr32NfPczz3Pfc3kyWau3KfITCRJkiRJekHRAUiSJEmSGoMJoiRJkiQJMEGUJEmSJJWZIEqSJEmSABNESZIkSVKZCaIkSZIkCTBBlCSJiBgVEb+LiP2LjkWSpCKF+yBKkoabiPhdxeGLgKeBTeXj92Xm5+sflSRJw58JoiRpWIuIh4D3ZObXt1Jnl8x8pn5R1U8zvzdJUv05xFSS1HQi4p8i4qaIuCEingTeFhFHRcQPI2J9RDwSEVdGREu5/i4RkRExoXz8ufL52yLiyYj4QURMHKStF0TEzRHx6/K1vxURr6g4/6KIuCIiHo6IDRHxnYh4YfncseWYNkTEmoh4e7n8zog4t+Ia74mIb/WL9S8jYjXws3L5VRHRHRG/jYi7I+LVFa/fJSL+MSJ+UT7fGRFjI+K/IuJf+r2f2yLigp3/U5AkDUcmiJKkZnUG8AVgNHAT8AzwQWAf4GjgZOB9W3n9W4B/BPYCHgb+71bqfgWYBLwM+Cnw2YpzVwDTgCPL1/p74Nlywvk/wMeAvYHpwMohvL/TgMOBqeXju8rt7AXcDPx3XyIKzAHeSOk9twHvAZ4CrgPeEhEvAIiIfYHjgBuHEIckqYmYIEqSmtWdmXlLZj6bmb2ZeXdm3pWZz2Tmg8BCSsnQYG7OzM7M3Ah8HjhsoErl61+bmU9m5lPAPODPImL3iBgFnAtcmJmPZOamzLyzfM23Abdn5qJyTI9m5j1DeH//nJlPZGZvOY7PZubj5eGmlwEvAQ4q130P8PeZ+UA53nvKdb8P9FZ8DmcDX8/MR4cQhySpiZggSpKa1ZrKg4g4OCL+pzwU9LfAfEq9iYP5dcXzPwAvHqhSeQXUyyLiwfJ1V5dP7QPsC+wK/GKAl44fpHx79X9/fxcRP4uIDcATwO489/621tb1lJJVyo+fHaSeJGkEMEGUJDWr/quw/Rel4Z8HZeZLgLlAVKGdc4DXASdQGs7a12sXwP8CfwRePsDr1gxSDvB7Squz9nnZAHU2v7+IeA3w18AbKA0h3RP4Hc+9v6219VngzIiYXq5zyyD1JEkjgAmiJGmk2APYAPy+vIjM1uYfDvW6TwOPUUrqPtp3IjM3AdcCH4+Il5V7G48uL47zOeDkiHhDeRGZfSLi0PJL7wHeEBGtEfEnwLu2I4ZngEeBFkrDXHevOP8p4J8i4uVRclhE7FWO8Vfl9q4D/rs8TFaSNEKZIEqSRoq/Ad4BPEmpN/GmKl33M8Da8s+9wPf7nf8QcD+wHHgc+GdK20z9EjgV+HC5/Mc8t+DM5ZR6CH8DXEMpmdyaW4GvAw8ADwG/BR6pOL8AWALcUT63ENit4vx15bYdXipJI5z7IEqSNMJFxAnAp4ED0y8GkjSi2YMoSdIIFhG7Utr+45Mmh5IkE0RJkkaoiJhKacXTvYArCw5HktQAHGIqSZIkSQLsQZQkSZIklZkgSpIkSZIA2KXoAOphn332yQkTJhQdhiRJkiQVYvny5Y9m5pht1RsRCeKECRPo7OwsOgxJkiRJKkRE/Gp76jnEVJIkSZIEmCBKkiRJkspMECVJkiRJQIMliBFxTUT8JiJ+Osj5iIgrI2J1RHRFxCvrHaMkSZIkNauGShCBa4GTt3L+tcCk8s95wH/WISZJkiRJGhEaKkHMzO8Aj2+lyunA9VnyQ6AtIvarT3SSJEmS1NyG2zYX44A1Fcfd5bJHiglHkiRpeFqyoocFy1axdn0vY9tamTNzMrOmjys6LGmnFHlfN8vfqeGWIMYAZTlgxYjzKA1DZf/9969lTJIkScPKkhU9XLx4Jb0bNwHQs76XixevBKj5F9pm+RI9FCPxPReh6Pu6qLarbbgliN3A+IrjdmDtQBUzcyGwEKCjo2PAJFKSJAmK+wJfVLsLlq3a/EW2T+/GTSxYtqqm7Rf9JbqIz7vo91yUIj7rou7rotuutoaag7gdlgLnlFczfRWwITMdXipJknZY3xf4nvW9JM99gV+yoqcp2wVYu753SOXVsrUv0bVW1Odd5HsuSlGfdVH3ddFtV1tDJYgRcQPwA2ByRHRHxLsj4vyIOL9c5VbgQWA18EngLwsKVZIkNYmivsAXmTiMbWsdUnm1FPkluqjPu8j3vGRFD0df+g0mXvQ/HH3pN+rynw9Q3Gdd1H1ddNvV1lAJYmaenZn7ZWZLZrZn5qcz8xOZ+Yny+czMD2TmyzNzamZ2Fh2zJEmqjqK+zBb1Bb7IxGHOzMm0tozaoqy1ZRRzZk6uabtFfoku6vMu6j2PxB7qou7rotuutoZKECVJ0shU5JfZor7AF5kszZo+jkvOnMq4tlYCGNfWyiVnTq35XKkiv0QX9XkX9Z5HYg91Ufd10W1X23BbpEaSJDWhIhd4mDNz8haLiEB9vsAX1W6fWdPH1f3La197RSzMU9TnXdR7LrqHuqh7u4j7uhHariYTREmStIUiVh8s8stsUV/gi0yWilTUl+giP+8i3vPYtlZ6Bvj7U68eahh593aziMzm3wGio6MjOzudrihJGl4aYUl+KP3Pf62HSh196TcG/DI7rq2V7110Qs3alZpVUX+X1bgiYnlmdmyrnnMQJUnaiqIWThlpS/I30wIPUiNopjlxqi+HmEqSNIgiN7guak5eUUM9HZImVV+zzIlTfZkgSpI0iCIXTilySf4i5y35ZVaSiuUQU0mSBlHkwikjbUl+SVJjMEGUJGkQRe5TV1Si5rwlSRrZHGIqSdIgit7LC0bOkvySpMZggihJ0iCKXjjFRE2SVG8miJIkbYVJmiRpJHEOoiRJkiQJsAdRkjRMLFnR4x55kiTVmAmiJKnhFblhvSRJI4lDTCVJDW9rG9ZLkqTqMUGUJDW8IjeslyRpJDFBlCQ1vCI3rJckDSNdi+CKKTCvrfTYtajoiIYdE0RJUsObM3MyrS2jtiir14b1kqRhomsR3HIhbFgDZOnxlgvrlyQ2SXJqgihJanizpo/jkjOnMq6tlQDGtbVyyZlTXaBGkvScO+bDxn5TDzb2lsprrejktIoabhXTiDgZ+DdgFPCpzLy03/kDgGuAMcDjwNsys7vugUpSgUbilg9uWC9J2qoNg6QEg5VX09aS02mza99+FTVUD2JEjAKuBl4LHAKcHRGH9Kt2OXB9Zk4D5gOX1DdKSSpW35YPPet7SZ7b8mHJip6iQ5MkNZImGfK43Ua3D628mopMTqusoRJE4AhgdWY+mJl/BG4ETu9X5xDgjvLzbw5wXpKamls+SJK2qYmGPG63GXOhpd/iZS2tpfJaKzI5rbJGSxDHAWsqjrvLZZV+Aryh/PwMYI+I2LsOsUlSQ3DLB0kaZoroyStyPl5Rps2GU6+E0eOBKD2eemV9hngWmZxWWaPNQYwByrLf8d8CV0XEucB3gB7gmeddKOI84DyA/fffv7pRSlKBxra10jNAMliPLR9G4txHSdopfT15fclaX08e1DZxaaIhj0MybXYxc/762rxjfukzHt1eSg6H2fxDaLwEsRsYX3HcDqytrJCZa4EzASLixcAbMnND/wtl5kJgIUBHR0f/JFOShq05Mydz8eKVWwwzrceWD31zH/va7Zv7CJgkStp+XYua4kv0ditq8ZLR7eXhpQOUqzaKSk6rrNGGmN4NTIqIiRGxK3AWsLSyQkTsExF9cV9MaUVTSRoxitrywbmPknbaSJwXV1RPXhMNeVR9NVQPYmY+ExEXAMsobXNxTWbeGxHzgc7MXAocD1wSEUlpiOkHCgtY0ohW5HDLIrZ8cO6jpJ1W9FYARfReFtWTV/SQx5HWU9xEGipBBMjMW4Fb+5XNrXh+M3BzveOSpEojcbhlkXMfJdVAEV/gi5wXV9RcwBlzt2wX6teTV9SQx6I+a1VFow0xlaRhYSQOt5wzczKtLaO2KKvH3EepqRW1T11RQz2L3AqgqFU9i1xZsygjcQXVJtJwPYiSNByMxOGWfT2jrmIqVUmRvSxFDfUssjetyN7LJlm8ZLuN1BVUm4QJoiTtgJE63LKIuY9S0ypyPl5RX+CLnBfnqp7142c9rDnEVJJ2gMMtpRooarhlUW0X2ctS5FDPabPhQz+FeetLj/XqWXNVz/rxsx7WTBAlaQcUtdWE1LSK3P5gJM7HG4lf4EfiXMCi+FkPa5HZ/HvId3R0ZGdnZ9FhSJKkwVwxZZAhaeNLvUzN2Hb/OYhQStLq9UXabQikESUilmdmx7bqOQdRkiQVr8jhliNxPl5f+yaEkvoxQZQkScUrclGLIts2SZPUYJyDKEmSilfknLiROB9PkgZhgihJkopX5KIWLqghSZu5SI0kSZIkNTkXqZE0YixZ0cOCZatYu76XsW2tzJk52e0mJEmSdoAJoqRhbcmKHi5evJLejZsA6Fnfy8WLVwKYJEqSJA2RcxAlDWsLlq3anBz26d24iQXLVhUUkdQEuhaV9gac11Z6rMdm9ZKkhmAPoqRhbe363iGVS8NKERuZ99+8fcOa0jG4aIskjQD2IEoa1sa2tQ6pXBo2+hK1DWuAfC5Rq3Vv3h3zn0sO+2zsLZVLkpqeCaKkYW3OzMm0tozaoqy1ZRRzZk4uKCKpSopK1DZ0D61cktRUHGIqaVjrW4jGVUzVdIpK1Ea3l3stByiXJDU9E0RJw96s6eNMCNV8ikrUZszdcg4iQEtrqVyS1PQabohpRJwcEasiYnVEXDTA+f0j4psRsSIiuiLidUXEKUlSTc2YW0rMKtUjUZs2G069EkaPB6L0eOqVLlAjSSNEQ/UgRsQo4GrgRKAbuDsilmbmfRXV/gFYlJn/GRGHALcCE+oerCRJtdSXkNV7FdO+tk0IJWlEaqgEETgCWJ2ZDwJExI3A6UBlgpjAS8rPRwNr6xqhJEn1YqImSaqzRksQxwGVEy66gSP71ZkHfDUi/grYHfiL+oQmSZIkSc2t0eYgxgBl2e/4bODazGwHXgd8NiKe9z4i4ryI6IyIznXr1tUgVEmSJElqLo2WIHYD4yuO23n+ENJ3A4sAMvMHwG7APv0vlJkLM7MjMzvGjBlTo3AlSU2vaxFcMQXmtZUea71RvSRJBWq0BPFuYFJETIyIXYGzgKX96jwMzACIiFdQShDtIpQkVV/XotKWDxvWAFl6vOVCk0RJUtNqqAQxM58BLgCWAfdTWq303oiYHxGnlav9DfDeiPgJcANwbmb2H4YqSdLOu2P+lvsBQun4jvnFxCNJUo012iI1ZOatlLauqCybW/H8PuDoesclSRqBNnQPrVySpGGuoXoQJUlqKKPbh1YuSdIwZ4IoSdJgZsyFltYty1paS+WSJDUhE0RJkgYzbTaceiWMHg9E6fHUK928XpLUtBpuDqIkSQPqWlRaHGZDd2mI54y59UnUps02IZQkjRgmiJKkxte33UTfiqJ9202AyZskSVXkEFNJUuNzuwlJkurCBFGS1PjcbkKSpLowQZQkNT63m5AkqS5MECVJjc/tJiRJqgsTRElS43O7CUmS6sJVTCVJw4PbTUiSVHP2IEqSJEmSABNESZIkSVKZCaIkSZIkCTBBlCQNRdciuGIKzGsrPXYtKjoiSZJURS5SI6kqlqzoYcGyVaxd38vYtlbmzJzMrOnjig5L1dS1CG65EDb2lo43rCkdg4vHSJLUJOxBlLTTlqzo4eLFK+lZ30sCPet7uXjxSpas6Ck6NFXTHfOfSw77bOwtlUuSpKZggihppy1YtorejZu2KOvduIkFy1YVFNEIUMRQzw3dQyuXJEnDjgmipJ22dn3vkMq1k/qGem5YA+RzQz1rnSSObh9auSRJGnZMEKUms2RFD0df+g0mXvQ/HH3pN+oyzHNsW+uQyrWTihrqOWMutPT7M21pLZVLkqSm0HAJYkScHBGrImJ1RFw0wPkrIuKe8s/PI2J9EXFKjaiouYBzZk6mtWXUFmWtLaOYM3NyTdsdsYoa6jltNpx6JYweD0Tp8dQrXaBGkqQm0lCrmEbEKOBq4ESgG7g7IpZm5n19dTLzQxX1/wqYXvdApQa1tbmAtVxRtO/armJaJ6Pby8NLByivtWmzTQglSWpiDZUgAkcAqzPzQYCIuBE4HbhvkPpnA/+nTrFJDa/IuYCzpo8zIayXGXO33G4CHOopSZKqotGGmI4DKv9bvLtc9jwRcQAwEfhGHeKShgXnAo4QDvWUJEk10mg9iDFAWQ5S9yzg5szcNNDJiDgPOA9g//33r050UoObM3MyFy9eucUwU+cCNimHekqSpBpotB7EbmB8xXE7sHaQumcBNwx2ocxcmJkdmdkxZsyYKoYoNa5Z08dxyZlTGdfWSgDj2lq55MypDv2UJEnSdmm0HsS7gUkRMRHooZQEvqV/pYiYDOwJ/KC+4UmNz7mAddS1qLS1xIbu0gIxM+baqydJkoa1hkoQM/OZiLgAWAaMAq7JzHsjYj7QmZlLy1XPBm7MzMGGn0pSbfVtVt+3UEzfZvVgkihJkoatqEWOVU7yPp+ZT1T94jugo6MjOzs7iw5DUjO5YsogW02Mhw/9tP7xSJIkbUVELM/Mjm3Vq9UcxJdR2sNwUXnj+4EWn5Gk4auozeolSZJqqCYJYmb+AzAJ+DRwLvBARPxzRLy8Fu1JUt0Ntil9PTarlyRJqpGarWJanh/46/LPM5QWlbk5Ii6rVZuSVDcz5pY2p6/kZvWSJGmYq8kiNRFxIfAO4FHgU8CczNwYES8AHgD+rhbtSlLd9C1E4yqmkiSpidRqFdN9gDMz81eVhZn5bEScUqM2Jam+3KxekiQ1mVoNMb0VeLzvICL2iIgjATLz/hq1KUmSJEnaCbVKEP8T+F3F8e/LZZIkSZKkBlWrBDEqN7HPzGep3XBWSZIkSVIV1CpBfDAiLoyIlvLPB4EHa9SWJEmSJKkKapUgng+8GugBuoEjgfNq1JYkSZIkqQpqMuwzM38DnFWLa0uSJEmSaqNW+yDuBrwb+FNgt77yzHxXLdqTJEmSJO28Wg0x/SzwMmAm8G2gHXiyRm1JkiRJkqqgVgniQZn5j8DvM/M64PXA1Bq1JakRdC2CK6bAvLbSY9eioiOSJEnSENVq64mN5cf1ETEF+DUwoUZtSSpa1yK45ULY2Fs63rCmdAwwbXZ92r9jPmzohtHtMGNufdqVJElqMrXqQVwYEXsC/wAsBe4D/qVGbUkq2h3zn0sO+2zsLZXXWl9yumENkM8lp/ZgSpIkDVnVexAj4gXAbzPzCeA7wIHVbkNSg9nQPbTyatpacmovoiRJ0pBUvQcxM58FLqj2dSU1sNHtQyuvpiKTU0mSpCZTqyGmX4uIv42I8RGxV99PjdqSVLQZc6GldcuyltZSea0VmZxKkiQ1mVoliO8CPkBpiOny8k9njdqSVLRps+HUK2H0eCBKj6deWZ8hnkUmp5IkSU2mJquYZubEHX1tRJwM/BswCvhUZl46QJ3ZwDwggZ9k5lt2tD1JVTJtdjFz/vradBVTSZKknVaTBDEizhmoPDOv38brRgFXAycC3cDdEbE0M++rqDMJuBg4OjOfiIiXVi9yqXqWrOhhwbJVrF3fy9i2VubMnMys6eOKDqs5FZWcSpIkNZla7YN4eMXz3YAZwI+BrSaIwBHA6sx8ECAibgROp7RNRp/3AleXV0klM39TraClalmyooeLF6+kd+MmAHrW93Lx4pUAJomSJElqWLUaYvpXlccRMRr47Ha8dBywpuK4GziyX50/KV/ze5SGoc7LzNt3PFqp+hYsW7U5OezTu3ETC5atMkGUJElSw6pVD2J/fwAmbUe9GKAs+x3vUr7W8UA78N2ImJKZ67e4UMR5wHkA+++//1DjlXbK2vW9QyqXJEmSGkGt5iDewnOJ3QuAQ4BF2/HSbmB8xXE7sHaAOj/MzI3ALyNiFaWE8e7KSpm5EFgI0NHR0T/JlGpqbFsrPQMkg2PbWgeoLUmSJDWGWvUgXl7x/BngV5m5PbtW3w1MioiJQA9wFtB/hdIlwNnAtRGxD6Uhpw/ufMhS9cyZOXmLOYgArS2jmDNzcoFRSZIkSVtXqwTxYeCRzHwKICJaI2JCZj60tRdl5jMRcQGwjNL8wmsy896ImA90ZubS8rmTIuI+YBMwJzMfq9H7kHZI3zxDVzGVJEnScBKZ1R99GRGdwKsz84/l412B72Xm4Vt/ZW10dHRkZ2dnEU1LkiRJUuEiYnlmdmyr3gtq1P4ufckhQPn5rjVqS5IkSZJUBbVKENdFxGl9BxFxOvBojdqSJEmSJFVBreYgng98PiKuKh93A+fUqC1JlboWwR3zYUM3jG6HGXNh2uyio5IkSdIwUJMEMTN/AbwqIl5MaZ7jk7VoR1I/XYvglgthY3mLjQ1rSsdgkihJkqRtqskQ04j454hoy8zfZeaTEbFnRPxTLdqSVOGO+c8lh3029pbKJUmSpG2o1RzE12bm+r6DzHwCeF2N2pLUZ8Mg240OVi5JkiRVqFWCOCoiXth3EBGtwAu3Ul9SNYxuH1q5JEmSVKFWCeLngDsi4t0R8W7ga8B1NWpLUp8Zc6GldcuyltZSuSRJkrQNtVqk5rKI6AL+AgjgduCAWrQlqULfQjSuYipJkqQdUKttLgB+DTwLzAZ+CXyxhm1J6jNttgmhJEmSdkhVE8SI+BPgLOBs4DHgJkrbXLymmu1IkiRJkqqv2j2IPwO+C5yamasBIuJDVW5D2m5LVvSwYNkq1q7vZWxbK3NmTmbW9HFFhyVJkiQ1pGovUvMGSkNLvxkRn4yIGZTmIEp1t2RFDxcvXknP+l4S6Fnfy8WLV7JkRU99AuhaBFdMgXltpceuRfVpV5IkSdpBVU0QM/NLmflm4GDgW8CHgH0j4j8j4qRqtiVty4Jlq+jduGmLst6Nm1iwbFXtG+9aBLdcCBvWAFl6vOVCk0RJkiQ1tJpsc5GZv8/Mz2fmKUA7cA9wUS3akgazdn3vkMqr6o75sLFfOxt7S+WSJElSg6rVPoibZebjmflfmXlCrduSKo1tax1SeVVt6B5auSRJktQAap4gSkWZM3MyrS2jtihrbRnFnJmTa9/46PahlUuSJEkNwARRTWvW9HFccuZUxrW1EsC4tlYuOXNqfVYxnTEXWvr1VLa0lsolSZKkBlXtbS6khjJr+rhitrXo26j+jvmlYaWj20vJoRvYS5IkqYE1XIIYEScD/waMAj6VmZf2O38usADo26vgqsz8VF2D1PDRtai4JG3abBNCSZIkDSsNlSBGxCjgauBEoBu4OyKWZuZ9/arelJkX1D1ADS99W030rSbat9UEmLhJkiRJA2i0OYhHAKsz88HM/CNwI3B6wTFpuHKrCUmSJGlIGi1BHAesqTjuLpf194aI6IqImyNifH1C07DjVhOSJEnSkDRaghgDlGW/41uACZk5Dfg6cN2AF4o4LyI6I6Jz3bp1VQ5Tw4JbTUiSJElD0mgJYjdQ2SPYDqytrJCZj2Xm0+XDTwJ/NtCFMnNhZnZkZseYMWNqEqwanFtNSJIkSUPSUIvUAHcDkyJiIqVVSs8C3lJZISL2y8xHyoenAffXN0TtiCUreliwbBVr1/cytq2VOTMn1377CbeakCRJkoakoRLEzHwmIi4AllHa5uKazLw3IuYDnZm5FLgwIk4DngEeB84tLGBtlyUrerh48Up6N24CoGd9LxcvXglQnyTRhFCSJEnaLpHZf4pf8+no6MjOzs6iwxixjr70G/Ss731e+bi2Vr530QkFRCRJkiSNLBGxPDM7tlWv0eYgqgmtHSA53Fq5JEmSpGKYIKrmxra1DqlckiRJUjFMEFVzc2ZOprVl1BZlrS2jmDNzckERSZIkSRpIQy1So+bUtxBN3VcxlSRJkjQkJoiqi1nTx5kQSpIkSQ3OIaaSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZSaIkiRJkiTABFGSJEmSVGaCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAmCXogNQ/SxZ0cOCZatYu76XsW2tzJk5mVnTxxUdliRJkqQGYYI4QixZ0cPFi1fSu3ETAD3re7l48UoAk0RJkiRJgENMR4wFy1ZtTg779G7cxIJlqwqKSJIkSVKjMUEcIdau7x1SuSRJkqSRp+ESxIg4OSJWRcTqiLhoK/XeGBEZER31jG+4GtvWOqTyqutaBFdMgXltpceuRfVpV5IkSdJ2a6gEMSJGAVcDrwUOAc6OiEMGqLcHcCFwV30jHL7mzJxMa8uoLcpaW0YxZ+bk2jfetQhuuRA2rAGy9HjLhSaJkiRJUoNpqAQROAJYnZkPZuYfgRuB0weo93+By4Cn6hnccDZr+jguOXMq49paCWBcWyuXnDm1PgvU3DEfNvYbyrqxt1QuSZIkqWE02iqm44A1FcfdwJGVFSJiOjA+M78SEX9bz+CGu1nTxxWzYumG7qGVS5IkSSpEo/UgxgBluflkxAuAK4C/2eaFIs6LiM6I6Fy3bl0VQ9SQjW4fWrkkSZKkQjRagtgNjK84bgfWVhzvAUwBvhURDwGvApYOtFBNZi7MzI7M7BgzZkwNQ9Y2zZgLLf0Ww2lpLZVLkiRJahiNliDeDUyKiIkRsStwFrC072RmbsjMfTJzQmZOAH4InJaZncWEq+0ybTaceiWMHg9E6fHUK0vlkiRJkhpGQ81BzMxnIuICYBkwCrgmM++NiPlAZ2Yu3foVtFVdi0oLw2zoLg3vnDG3fknatNkmhJIkSVKDa6gEESAzbwVu7Vc24FjEzDy+HjE1hb6tJvpWE+3bagJM3CRJkiQBjTfEVLXiVhOSJEmStsEEcaRwqwlJkiRJ22CCOFK41YQkSZIHivL2AAAgAElEQVSkbTBBHCncakKSJEnSNpggjhRuNSFJkiRpGxpuFVPVkFtNSJIkSdoKexAlSZIkSYAJoiRJkiSpzARRkiRJkgSYIEqSJEmSykwQJUmSJEmACaIkSZIkqcwEUZIkSZIEmCBKkiRJkspMECVJkiRJgAmiJEmSJKnMBFGSJEmSBJggSpIkSZLKTBAlSZIkSUADJogRcXJErIqI1RFx0QDnz4+IlRFxT0TcGRGHFBGnJEmSJDWbhkoQI2IUcDXwWuAQ4OwBEsAvZObUzDwMuAz4WJ3DlCRJkqSm1FAJInAEsDozH8zMPwI3AqdXVsjM31Yc7g5kHeOTJEmSpKa1S9EB9DMOWFNx3A0c2b9SRHwA+GtgV+CE+oQmSZIkSc2t0XoQY4Cy5/UQZubVmfly4MPAPwx4oYjzIqIzIjrXrVtX5TAlSZIkqfk0WoLYDYyvOG4H1m6l/o3ArIFOZObCzOzIzI4xY8ZUMURJkiRJak6NliDeDUyKiIkRsStwFrC0skJETKo4fD3wQB3jkyRJkqSm1VBzEDPzmYi4AFgGjAKuycx7I2I+0JmZS4ELIuIvgI3AE8A7iotYkiRJkppHQyWIAJl5K3Brv7K5Fc8/WPegJEmSJGkEaLQhppIkSZKkgpggSpIkSZIAE0RJkiRJUlnDzUGUJEnSyLJx40a6u7t56qmnig5FGvZ222032tvbaWlp2aHXmyAWYMmKHhYsW8Xa9b2MbWtlzszJzJo+ruiwJEmSCtHd3c0ee+zBhAkTiIiiw5GGrczkscceo7u7m4kTJ+7QNRxiWmdLVvRw8eKV9KzvJYGe9b1cvHglS1b0FB2aJElSIZ566in23ntvk0NpJ0UEe++99071xpsg1tmCZavo3bhpi7LejZtYsGxVQRFJkiQVz+RQqo6d/btkglhna9f3DqlckiRJtfeud72Ll770pUyZMmXQOpnJhRdeyEEHHcS0adP48Y9/vPncddddx6RJk5g0aRLXXXfdNtv71re+xSmnnFKV2HfEtddey9q1awtrv9puv/12Jk+ezEEHHcSll146YJ2PfexjHHLIIUybNo0ZM2bwq1/9avO5UaNGcdhhh3HYYYdx2mmnbbO9hx56aKv3Sq0tWbKE++67rybXNkGss7FtrUMqlyRJUu2de+653H777Vutc9ttt/HAAw/wwAMPsHDhQt7//vcD8Pjjj/ORj3yEu+66ix/96Ed85CMf4YknntihOJ555pkdet1QNVOCuGnTJj7wgQ9w2223cd9993HDDTcMmDxNnz6dzs5Ourq6eOMb38jf/d3fbT7X2trKPffcwz333MPSpUt3OJZ6/fmZIDaROTMn09oyaouy1pZRzJk5uaCIJEmShpclK3o4+tJvMPGi/+HoS79RlbUcjj32WPbaa6+t1vnyl7/MOeecQ0Twqle9ivXr1/PII4+wbNkyTjzxRPbaay/23HNPTjzxxAGTzdtvv52DDz6YY445hsWLF28unzdvHueddx4nnXQS55xzDk899RTvfOc7mTp1KtOnT+eb3/wmUErqTj/9dE4++WQmT57MRz7ykc3X+NjHPsaUKVOYMmUKH//4x4Hn93JdfvnlzJs3j5tvvpnOzk7e+ta3cthhh9HbW+eRbF2L4IopMK+t9Ni1aKcu96Mf/YiDDjqIAw88kF133ZWzzjqLL3/5y8+r95rXvIYXvehFALzqVa+iu7t7SO0sX76cQw89lKOOOoqrr756c/m1117Lm970Jk499VROOukkMpM5c+YwZcoUpk6dyk033QSUeo2PPfZYzjjjDA455BDOP/98nn32WQBuuOEGpk6dypQpU/jwhz+8+dovfvGLNz+/+eabOffcc/n+97/P0qVLmTNnDocddhi/+MUvhvQ+tsVVTOusb7VSVzGVJEkaur4F//rWdOhb8A+o+fepnp4exo8fv/m4vb2dnp6eQcsrPfXUU7z3ve/lG9/4BgcddBBvfvObtzi/fPly7rzzTlpbW/nXf/1XAFauXMnPfvYzTjrpJH7+858DpWTopz/9KS960Ys4/PDDef3rX09E8JnPfIa77rqLzOTII4/kuOOOY8899xzwfbzxjW/kqquu4vLLL6ejo6Mqn81261oEt1wIG8tJ6YY1pWOAabN36JIDff533XXXVl/z6U9/mte+9rWbj5966ik6OjrYZZdduOiii5g1a9bzXvPOd76Tf//3f+e4445jzpw5W5z7wQ9+QFdXF3vttRdf/OIXueeee/jJT37Co48+yuGHH86xxx4LlP787rvvPg444ABOPvlkFi9ezKtf/Wo+/OEPs3z5cvbcc09OOukklixZMmAMAK9+9as57bTTOOWUU3jjG9+43Z/T9jJBLMCs6eNMCCVJknbA1hb8q/X3q8x8XllEDFpe6Wc/+xkTJ05k0qRJALztbW9j4cKFm8+fdtpptLaWphzdeeed/NVf/RUABx98MAcccMDmBPHEE09k7733BuDMM8/kzjvvJCI444wz2H333TeXf/e7392uuXR1d8f855LDPht7S+U7mCBuz+df6XOf+xydnZ18+9vf3lz28MMPM3bsWB588EFOOOEEpk6dystf/vLN5zds2MD69es57rjjAHj729/Obbfdtvl8Xw8ylP78zj77bEaNGsW+++7Lcccdx913381LXvISjjjiCA488EAAzj77bO68805aWlo4/vjjGTNmDABvfetb+c53vjNoglhrDjEtQpW71SVJkkaKIhf8a29vZ82aNZuPu7u7GTt27KDl/W0taelL7mDghGewawyWoALssssum4cwAju19UHVbBhkWOdg5dthez9/gK9//et89KMfZenSpbzwhS/cXN5X/8ADD+T4449nxYoVW7wuM+v+59e/fr3+/EwQ662vW33DGiCf61Y3SZQkSdqmIhf8O+2007j++uvJTH74wx8yevRo9ttvP2bOnMlXv/pVnnjiCZ544gm++tWvMnPmzC1ee/DBB/PLX/5y83yxG264YdB2jj32WD7/+c8D8POf/5yHH36YyZNL61V87Wtf4/HHH6e3t5clS5Zw9NFHc+yxx7JkyRL+8Ic/8Pvf/54vfelL/Pmf/zn77rsvv/nNb3jsscd4+umn+cpXvrK5jT322IMnn3yy2h/Rto1uH1r5djj88MN54IEH+OUvf8kf//hHbrzxxgF7T1esWMH73vc+li5dyktf+tLN5U888QRPP/00AI8++ijf+973OOSQQ7Z4bVtbG6NHj+bOO+8E2PznM5Bjjz2Wm266iU2bNrFu3Tq+853vcMQRRwClIaa//OUvefbZZ7nppps45phjOPLII/n2t7/No48+yqZNm7jhhhs291Tuu+++3H///Tz77LN86Utf2txGLf/8TBDrbWvd6pIkSdqqWi34d/bZZ3PUUUexatUq2tvb+fSnPw3AJz7xCT7xiU8A8LrXvY4DDzyQgw46iPe+9738x3/8BwB77bUX//iP/8jhhx/O4Ycfzty5c5+34M1uu+3GwoULef3rX88xxxzDAQccMGgsf/mXf8mmTZuYOnUqb37zm7n22ms393Ydc8wxvP3tb+ewww7jDW94Ax0dHbzyla/k3HPP5YgjjuDII4/kPe95D9OnT6elpYW5c+dy5JFHcsopp3DwwQdvbuPcc8/l/PPPr/8iNTPmQku/ZL6ltVS+g3bZZReuuuoqZs6cySte8Qpmz57Nn/7pnwIwd+7czauSzpkzh9/97ne86U1v2mI7i/vvv5+Ojg4OPfRQXvOa13DRRRc9L0EE+MxnPsMHPvABjjrqqM3DgQdyxhlnMG3aNA499FBOOOEELrvsMl72spcBcNRRR3HRRRcxZcoUJk6cyBlnnMF+++3HJZdcwmte8xoOPfRQXvnKV3L66acDcOmll3LKKadwwgknsN9++21u46yzzmLBggVMnz696ovUxNa6NJtFR0dHdnZ2Fh1Gybw2YKDPPGDe+npHI0mSVLj777+fV7ziFdtdf8mKnhG54N+1115LZ2cnV111VdGh7JyuRaXOkQ3dpZ7DGXN3eP7hcPKtb32Lyy+/fIue3FoZ6O9URCzPzG2uSuQiNfU2ur08vHSAckmSJG2TC/4Nc9Nmj4iEcLhyiGm91aBbXZIkSc3v3HPPHf69hyPY8ccfX5few51lglhv02bDqVfC6PFAlB5PvdL/RZEkSZJUOIeYFsFudUmSpC1saxsBSdtnZ9eYabgexIg4OSJWRcTqiLhogPN/HRH3RURXRNwREYMvASVJkqSGt9tuu/HYY4/t9BdbaaTLTB577DF22223Hb5GQ/UgRsQo4GrgRKAbuDsilmbmfRXVVgAdmfmHiHg/cBnw5vpHK0mSpGpob2+nu7ubdevWFR2KNOzttttutLfv+AKYDZUgAkcAqzPzQYCIuBE4HdicIGbmNyvq/xB4W10jlCRJUlW1tLQwceLEosOQROMNMR0HVO4B0V0uG8y7gdtqGpEkSZIkjRCN1oM40MzkAQejR8TbgA7guEHOnwecB7D//vtXKz5JkiRJalqN1oPYDYyvOG4H1vavFBF/Afz/wGmZ+fRAF8rMhZnZkZkdY8aMqUmwkiRJktRMopFWi4qIXYCfAzOAHuBu4C2ZeW9FnenAzcDJmfnAdl53HfCr6ke80/YBHi06CDUt7y/VkveXasn7S7XmPaZaatT764DM3GbPWUMliAAR8Trg48Ao4JrM/GhEzAc6M3NpRHwdmAo8Un7Jw5l5WkHh7pSI6MzMjqLjUHPy/lIteX+plry/VGveY6ql4X5/NdocRDLzVuDWfmVzK57/Rd2DkiRJkqQRoNHmIEqSJEmSCmKCWKyFRQegpub9pVry/lIteX+p1rzHVEvD+v5quDmIkiRJkqRi2IMoSZIkSQJMEAsRESdHxKqIWB0RFxUdj5pPRDwUESsj4p6I6Cw6Hg1vEXFNRPwmIn5aUbZXRHwtIh4oP+5ZZIwavga5v+ZFRE/5d9g95RXOpSGLiPER8c2IuD8i7o2ID5bL/R2mnbaV+2tY/w5ziGmdRcQoSns9ngh0U9rr8ezMvK/QwNRUIuIhoCMzG3EPHg0zEXEs8Dvg+sycUi67DHg8My8t/0fXnpn54SLj1PA0yP01D/hdZl5eZGwa/iJiP2C/zPxxROwBLAdmAefi7zDtpK3cX7MZxr/D7EGsvyOA1Zn5YGb+EbgROL3gmCRpUJn5HeDxfsWnA9eVn19H6R9EacgGub+kqsjMRzLzx+XnTwL3A+Pwd5iqYCv317Bmglh/44A1FcfdNMGNpIaTwFcjYnlEnFd0MGpK+2bmI1D6BxJ4acHxqPlcEBFd5SGoDv/TTouICcB04C78HaYq63d/wTD+HWaCWH8xQJnjfFVtR2fmK4HXAh8oD+GSpOHiP4GXA4cBjwD/Wmw4Gu4i4sXAF4H/LzN/W3Q8ai4D3F/D+neYCWL9dQPjK47bgbUFxaImlZlry4+/Ab5EaWizVE3/W5570TcH4zcFx6Mmkpn/m5mbMvNZ4JP4O0w7ISJaKH15/3xmLi4X+ztMVTHQ/TXcf4eZINbf3cCkiJgYEbsCZwFLC45JTSQidi9PlCYidgdOAn669VdJQ7YUeEf5+TuALxcYi5pM3xf3sjPwd5h2UEQE8Gng/sz8WMUpf4dppw12fw3332GuYlqA8lK3HwdGAddk5kcLDklNJCIOpNRrCLAL8AXvMe2MiLgBOB7YB/hf4P8AS4BFwP7Aw8CbMtOFRjRkg9xfx1MampXAQ8D7+uaLSUMREccA3wVWAs+Wi/+e0jwxf4dpp2zl/jqbYfw7zARRkiRJkgQ4xFSSJEmSVGaCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJ0naKiE0RcU/Fz0VVvPaEiBhWe2VJkprPLkUHIEnSMNKbmYcVHYQkSbViD6IkSTspIh6KiH+JiB+Vfw4qlx8QEXdERFf5cf9y+b4R8aWI+En559XlS42KiE9GxL0R8dWIaC3sTUmSRiQTREmStl9rvyGmb64499vMPAK4Cvh4uewq4PrMnAZ8HriyXH4l8O3MPBR4JXBvuXwScHVm/imwHnhDjd+PJElbiMwsOgZJkoaFiPhdZr54gPKHgBMy88GIaAF+nZl7R8SjwH6ZubFc/khm7hMR64D2zHy64hoTgK9l5qTy8YeBlsz8p9q/M0mSSuxBlCSpOnKQ54PVGcjTFc834VoBkqQ6M0GUJKk63lzx+IPy8+8DZ5WfvxW4s/z8DuD9ABExKiJeUq8gJUnaGv9nUpKk7dcaEfdUHN+emX1bXbwwIu6i9J+vZ5fLLgSuiYg5wDrgneXyDwILI+LdlHoK3w88UvPoJUnaBucgSpK0k8pzEDsy89GiY5EkaWc4xFSSJEmSBNiDKEmSJEkqswdRkqSyiJgQERkRztGXJI1IJoiSpKYREcsiYv4A5adHxK9N/CRJ2joTRElSM7kWeHtERL/ytwOfz8xn6h9SdUSJ/25LkmrKf2gkSc1kCbAX8Od9BRGxJ3AKcH35+PURsSIifhsRayJi3vZePCIuiohfRMSTEXFfRJzR7/x7I+L+ivOvLJePj4jFEbEuIh6LiKvK5fMi4nMVr99iiGtEfCsiPhoR3wP+ABwYEe+saOPBiHhfvxhOj4h7yu/vFxFxckS8KSKW96v3NxGxZHvfuyRpZDBBlCQ1jczsBRYB51QUzwZ+lpk/KR//vny+DXg98P6ImLWdTfyCUvI5GvgI8LmI2A8gIt4EzCtf+yXAacBjETEK+ArwK2ACMA64cQhv6+3AecAe5Wv8hlLC+xJK+ypeUZGIHkEpEZ5Tfn/HAg8BS4GJEfGKiuu+DfjsEOKQJI0AJoiSpGZzHfCmiGgtH59TLgMgM7+VmSsz89nM7AJuAI7bngtn5n9n5trya28CHgCOKJ9+D3BZZt6dJasz81fl82OBOZn5+8x8KjPvHML7uTYz783MZzJzY2b+T2b+otzGt4Gv8lyP6buBazLza+UYezLzZ5n5NHATpaSQiPhTSsnqV4YQhyRpBDBBlCQ1lXLytQ44PSIOBA4HvtB3PiKOjIhvlod7bgDOB/bZnmtHxDnl4ZvrI2I9MKXiteMp9TD2Nx741U7Mf1zTL4bXRsQPI+Lxcgyv244YoJQkv6U8P/PtwKJy4ihJ0mYmiJKkZnQ9pZ7DtwNfzcz/rTj3BUpDLsdn5mjgE0D/RW2eJyIOAD4JXADsnZltwE8rXrsGePkAL10D7D/ICqq/B15UcfyyAeps3rA4Il4IfBG4HNi3HMOt2xEDmflD4I+UehvfgsNLJUkDMEGUJDWj64G/AN5LxfDSsj2AxzPzqfKcvbds5zV3p5SsrQOIiHdS6kHs8yngbyPiz8orjh5UTip/BDwCXBoRu0fEbhFxdPk19wDHRsT+ETEauHgbMewKvLAcwzMR8VrgpIrznwbeGREzIuIFETEuIg6uOH89cBXwzBCHuUqSRggTRElS08nMh4DvU0rqlvY7/ZfA/Ih4EphLaVGb7bnmfcC/Aj8A/heYCnyv4vx/Ax+l1EP5JOUVVTNzE3AqcBDwMNANvLn8mq9RmhvYBSxnG3MCM/NJ4MJyzE9QSm6XVpz/EeWFa4ANwLeBAyou8VlKSa29h5KkAUVmbruWJEka9soL9/wGeGVmPlB0PJKkxmMPoiRJI8f7gbtNDiVJgxlowrwkSWoyEfEQpcVstnfPR0nSCOQQU0mSJEkS4BBTSZIkSVKZCaIkSZIkCRghcxD32WefnDBhQtFhSJIkSVIhli9f/mhmjtlWvRGRIE6YMIHOzs6iw5AkSZKkQkTEr7annkNMJUmSJEmACaIkSZIkqcwEUZIkSZIEmCBKkiRJkspqmiBGxMkRsSoiVkfERQOcPz8iVkbEPRFxZ0QcUi6fEBG95fJ7IuITFa/5s/JrVkfElRERtXwPkiRJkjRS1CxBjIhRwNXAa4FDgLP7EsAKX8jMqZl5GHAZ8LGKc7/IzMPKP+dXlP8ncB4wqfxzcq3egyRJkiSNJLXsQTwCWJ2ZD2bmH4EbgdMrK2TmbysOdwdyaxeMiP2Al2TmDzIzgeuBWdUNW5IkFaJrEVwxBea1lR67FhUdkSSNOLXcB3EcsKbiuBs4sn+liPgA8NfArsAJFacmRsQK4LfAP2Tmd8vX7O53zXFVjluSJNVb1yK45ULY2Fs63rCmdAwwbXZxcUnSCFPLHsSB5gY+r4cwM6/OzJcDHwb+oVz8CLB/Zk6nlDx+ISJesr3XBIiI8yKiMyI6161bt0NvQJKkEamInrw75j+XHPbZ2FsqlyTVTS0TxG5gfMVxO7B2K/VvpDxcNDOfzszHys+XA78A/qR8zfbtuWZmLszMjszsGDNmzA6/CUmSRpS+nrwNa4B8riev1knihu6hlUuSaqKWCeLdwKSImBgRuwJnAUsrK0TEpIrD1wMPlMvHlBe5ISIOpLQYzYOZ+QjwZES8qrx66TnAl2v4HiRJGlmK6skb3T60cklSTdQsQczMZ4ALgGXA/cCizLw3IuZHxGnlahdExL0RcQ+loaTvKJcfC3RFxE+Am4HzM/Px8rn3A58CVlPqWbytVu9BkqQRp6ievBlzoaV1y7KW1lK5JKluarlIDZl5K3Brv7K5Fc8/OMjrvgh8cZBzncCUKoYpSZL6jG4vDy8doLyW+haiuWN+KRkd3V5KDl2gRpLqqqYJoiRJGmZmzN1yNVGoX0/etNkmhJJUsFrOQZQkScPNtNlw6pUwejwQpcdTrzRxk6QRwh5ESZK0JXvyJGnEsgdRkiRJkgSYIEqSJI1MXYvgiikwr630WOu9LiUNCw4xlSRJGmm6Fm25GNGGNaVjcHixNMLZgyhJkjTS3DF/y5VqoXR8x/xi4pHUMEwQJUmSRpoN3UMrlzRimCBKkiSNNKPbh1YuacQwQZT+X3v3H2dlWSf+//UWUIZswR9kyqBi8sUICPYzaqWLlilaimRG0g+zLLPy67Y9lsTvbq7y3T6y6lYfVreN3dLaTCVToi2lVrOiNWMIFn9lmpbOYKuoWBYowvvzxzmDB5wZZpi552bOeT0fj3mcc1/3r+u+5+Zw3nNd1/uSJKnRHHcRDGvatmxYU6VcUkMzQJQkSWo0U2bDKQth5FggKq+nLDRBjSSzmEqSJDWkKbMNCCW9jC2IkiRJkiTAAFGSJEmSVGWAKEmSJEkCDBAlSZIkSVUGiJIkSZIkwABRkiRJklRlgChJkiRJAgwQJUmSJKnv1iyGz0+Ci0dVXtcsLrtGO2Vo2RWQJEmSpEFtzWL4zvmwaUNl+dnHKssAU2aXV6+dYAuiJEmSBk6dtLJI27ht/kvBYYdNGyrlg0yhAWJEnBgRD0TEQxExr5P150bE3RGxOiKWR8TEavnxEbGyum5lRLylZp87qsdcXf15VZHXIEmSpH7S0cry7GNAvtTKYpCowe7Ztt6V78IKCxAjYghwFXASMBGY0xEA1vhGZk7OzKnAZcDnquXrgFMyczLwAeDft9vvvZk5tfrzRFHXIEmSpH5UR60svWKraf0b2dy78l1YkS2IRwAPZebDmfkCcD1wau0Gmfn7msVXAFktX5WZa6vl9wLDI2KPAusqSZKkotVRK0uP2WraGI67CIY1bVs2rKlSPsgUGSCOAR6rWW6rlm0jIj4REb+m0oJ4fifHeSewKjOfrym7utq99DMREf1ZaUmSJBWkjlpZeqxRW00bzZTZcMpCGDkWiMrrKQsHXYIaKDaLaWeBW76sIPMq4KqIeA/wt1S6lFYOEPE64B+AE2p2eW9mtkfEK4FvAe8Hvvayk0ecA5wDcOCBB/bhMiRJktQvjrto20yPMGhbWXqsEVtNG9WU2YMyINxekS2IbcDYmuVmYG0X20KlC+qsjoWIaAZuBs7MzF93lGdme/X1D8A3qHRlfZnMXJSZLZnZMnr06J2+CEmSJPWTsltZyhgL2IitphrUimxBXAGMj4hxQDtwBvCe2g0iYnxmPlhdfDvwYLV8FPBd4MLM/GnN9kOBUZm5LiKGAScD/1ngNUiSJKk/ldXKUtY8dY3YalqmNYsr3XefbasE4cddVBetegOpsBbEzHwROA9YBtwPLM7MeyNifkTMrG52XkTcGxGrgU/xUvfS84BDgc9sN53FHsCyiFgDrKYSeP5rUdcgSZKkOlHWWMCyW00biQmB+kVkvmxYYN1paWnJ1tbWsqshSZKkslw8ik7SYQABF68f6NqoCJ+fVA0OtzNyLPzVPQNfn11MRKzMzJYdbVfkGERJkiRp1+BYwPpnQqB+YYAoSZKk+ldH89SpC/4RoF8YIEqSJGnALFnVzlELbmfcvO9y1ILbWbKqfWBOPGU2KyZfwu8YzZYMfsdoVky+xLGA9cQ/AvSLIrOYSpIkSVstWdXOhTfdzYZNmwFoX7+BC2+6G4BZ08YUf+4VB7Fh0//ZWta0YgiXjm0v/NwaIB3BvllM+8QAUZIkSQPi8mUPbA0OO2zYtJnLlz1QeJBW5rk1gOpksvoy2cVUkiSpjAnUG9Da9Rt6VV4v55YGEwNESZLU2Jw7bcAcMKqpV+X1cm5pMDFAlCRJja2sCdQb0NwZE2gaNmSbsqZhQ5g7Y0Jdn1saTByDKEmSGptzpw2YjrF+ly97gLXrN3DAqCbmzpgwIGMAyzy3NJgYIEqSpMY2srnavbST8jq2ZFV7aYFaWUFZmeeWBgu7mErSzjKphVQfGnDutI7pJtrXbyB5abqJAZuTUNIuywBRknaGSS2k+jFlNpyyEEaOBaLyesrCuk6V392UD5Iam11MJWlndJfUoo6/VEp1q8HmTnPKB0ldMUCUpJ1hUgvVsbLGpjWisu71AaOaaO8kGHTKB0l2MZWkndFV8oo6T2qh+ufYtIFT5r12ygdJXTFAlKSd0YBJLTTwlqxq56gFtzNu3nc5asHtAxI4ODZt4Fy+7AGO3/wjlrieKV8AACAASURBVO9+Pg/v8R6W734+x2/+0YDc61nTxnDpaZMZM6qJAMaMauLS0ybbUizJLqaStFM6xirdNr/SrXRkcyU4bKAxTCpWR+tSR7DW0boEFPol3rFpA6fl9z/g0mH/xoh4AYDmWMeCYf/Ghb8HeEvh53fKhwawZnF5/0+VeW71iQGiJO2sBktqoYHVXUtekV/qHZs2cC7c/ZuM4IVtykbEC1y4+zeBS8uplOpHR7btjoRqHdm2ofj/u8o8t/rMLqaSJO2CymrJc2zawNmPdb0ql3qlu2zbdXzuMrrm1xsDREmSdkFdtdgV3ZLn2LSBE10kteqqXOqVMrNtl3Ruk2z1D7uYSpIGhwYbzzJ3xgSW3/zPfJLrOSDWsTb35QucwdEzPl74uRtxbFop000cd9G23fDAZFfqPyObK107Oyuv03OX1TW/3tiCKEna9XWMZ3n2MSBfGs+yZnHhpy6ru9KsIT9lwbB/o3m3dewW0LxbJYHJrCE/HZDzN5LSWh2mzIZTFsLIsUBUXk9ZWNd/+NAAKjPbdknnNslW/7AFUZK06+tuPEuBX6bLyiQKwG3zGbp54zZFQzdvLPyaG1GprQ4mu1JRysy2XdK5TbLVPwoNECPiROD/AEOAf8vMBdutPxf4BLAZeA44JzPvq667EDi7uu78zFzWk2NKkupQSeNZSg0cyhw/1GBsdVDdKvMPECWce+6MCdv8UQ9MsrUzCutiGhFDgKuAk4CJwJyImLjdZt/IzMmZORW4DPhcdd+JwBnA64ATgX+OiCE9PKYkqd50NW6l4PEspQYOJV1zIyorIZCk/mWSrf5RZAviEcBDmfkwQERcD5wK3NexQWb+vmb7VwBZfX8qcH1mPg88EhEPVY/Hjo4pNbpSEi2UrcGSlzSkkpJ5lNpdyQQmA8ZWB6l+NGKSrf5WZJKaMUBt+qK2atk2IuITEfFrKi2I5+9g3x4dU2pUDZneucTkJRpAJSXzKHVOQBOYDBhbHSTpJUW2IEYnZfmygsyrgKsi4j3A3wIf6GbfzgLalx0TICLOAc4BOPDAA3tYZakfldCq1ZDpnUtKXqISlDCepePfTWmt8iYwGTC2OkhSRZEBYhswtma5GVjbzfbXA1/swb49OmZmLgIWAbS0tHQaREqF6WjV6ghcOlq1oNAve2vXb2Dmbsv59NDFW+dNu+zF2Xxn/dGFnbN0JvIYWA3YndfAQZLUSIrsYroCGB8R4yJidypJZ5bWbhAR42sW3w48WH2/FDgjIvaIiHHAeODnPTmmtEvorlWrQB/Y8+edzpv2gT1/Xuh5S2Uij4Fjd15JkupeYS2ImfliRJwHLKMyJcVXMvPeiJgPtGbmUuC8iHgrsAl4hkr3UqrbLaaSfOZF4BOZuRmgs2MWdQ3STiupVevTw25gxIsvbFM2Il7g08NuAC4p9NylMZHHwLE7ryTtlLISyDVk4roS1cv9LnQexMz8HvC97couqnn/l93s+1ngsz05prTLGdlcbWXppLxAIzb8rlfldaHMiYAbjd15JanXOhLIdeQI6EggBxQaPJR13kZVT/e7yC6mUuM67qJKK1atgWjVatTullNmw1/dAxevr7waHBajUZ8vSeqD7hLI1eN5G1U93W8DRKkIZaWnLyswVWPw+ZKkXlvbyVyq3ZUP9vM2qnq634V2MZUaWhnp6e1uqSL5fElSrx0wqon2ToKEA0Y1dbL14D9vo6qn+22AKNWbsuZNa8DpD0pV1v12Xj5J6pW5MyZsMzYNoGnYEObOmFCX521U9XS/DRAl9V1J8z42LO+3JA0aHQlKBjq7ZVnnbVT1dL8js/7nkG9pacnW1tayqyHVr89P6iJr69hK0hj1L++3JEnqpYhYmZktO9rOFkRJfef0BwPL+606VS9ziEnSYGYWU0l95/QHA8v7rTrUMYdY+/oNJC/NIbZkVXvZVZOkhmKAKKnvnP5gYHm/VYfqaQ4xSRrMDBAl9V1Z8z42Ku+36lA9zSEmSYOZYxAl9Q+nPxhY3m/VmXqaQ0ySY4oHM1sQJUlS6ebOmEDTsCHblA3WOcSkRueY4sHNAFGSpO6sWVyZWuTiUZXXNYvLrlFdmjVtDJeeNpkxo5oIYMyoJi49bbItDtIg5Jjiwc0uppIkdWXNYvjO+bCp2vXx2ccqy2AX3wLMmjbGgFCqA44pHtxsQZQkqSu3zX8pOOywaUOlXJLUqa7GDjumeHAwQJTUL5asaueoBbczbt53OWrB7Y4zUH14tq135ZIkxxQPcnYxldRnHYPRO8YbdAxGB+wupsFtZHOlW2ln5ZKkTnX8328W08HJAFFSn3U3GN3/DDSoHXfRtmMQAYY1VcolSV1yTPHgZRdTSX3mYHTVrSmz4ZSFMHIsEJXXUxaaoEaSVLdsQZTUZ05wrbo2ZbYBoSSpYdiCKKnPHIwuSZJUH3bYghgR5wHXZuYzA1AfqV8tWdXuAOkB4GB0DQT/PUuSVLyedDF9NbAiIn4BfAVYlplZbLVUb8r4YrdkVTvLb/5nbuB6DthjHWv/tC9fuPkM4ON+qSyAg9FVJDPlSpI0MHbYxTQz/xYYD3wZOAt4MCL+d0S8Zkf7RsSJEfFARDwUEfM6Wf+piLgvItZExG0RcVC1/M0RsbrmZ2NEzKquuyYiHqlZN7WX16wB1vHFrn39BpKXvtgVPU/e6u8uYn4sonm3dewW0LzbOubHIlZ/d1Gh55XU/7rLlCtJkvpPj8YgVlsMf1f9eRHYC7gxIi7rap+IGAJcBZwETATmRMTE7TZbBbRk5hTgRuCy6vl+mJlTM3Mq8BbgT8D3a/ab27E+M1f35BpUnrK+2H34ha8zIl7YpmxEvMCHX/h6oeeVBsKSVe0cteB2xs37LkctuL3wP7iUzUy5kiQNjB0GiBFxfkSspBK8/RSYnJkfA/4X8M5udj0CeCgzH87MF4DrgVNrN6gGgn+qLv4M6Gzm4dOBW2q20yBT1he7A3Z7qlfl0mBRVqt8mbrKiGumXEmS+ldPWhD3BU7LzBmZ+c3M3ASQmVuAk7vZbwzwWM1yW7WsK2cDt3RSfgZw3XZln612S/18ROyxwytQqcr6Yrex6dW9KpcGi0bsbmmmXEmSBkZPAsTvAU93LETEKyPiSIDMvL+b/aKTsk6T20TE+4AW4PLtyvcHJgPLaoovBA4DDgf2Bi7o4pjnRERrRLQ++eST3VRTRSvri92Ik+bz4pDh25S9OGQ4I06aX+h5paI1YnfLWdPGcOlpkxkzqokAxoxq4tLTJpugRpKkftaTLKZfBP68ZvmPnZR1pg0YW7PcDKzdfqOIeCvwN8Axmfn8dqtnAzd3tFoCZObj1bfPR8TVwF93dvLMXAQsAmhpaTHraolKmwJhyuzKA37bfHi2DUY2M/S4i5zwWoPeAaOaaO8kGKz37pZmypUkqXg9CRCjdlqLzNwSET3ZbwUwPiLGAe1Uuoq+Z5sDR0wDvgScmJlPdHKMOVRaDGv32T8zH4+IAGYB9/SgLipZaV/spswuLSB0zjYVZe6MCdtM+QB2t5QkSf2jJ4HewxFxPpVWQ4CPAw/vaKfMfDEizqPSPXQI8JXMvDci5gOtmbmUSpfSPYFvVuI9Hs3MmQARcTCVFsgfbXfoayNiNJUurKuBc3twDdKAcs42Fam0VnlJklT3Ykdz3kfEq4CFVKabSOA24JNdtPjtklpaWrK1tbXsaqiBHLXg9k67AI4Z1cRP572lhBqpCLYSS5KkwSIiVmZmy46222ELYjUQPKNfaiU1iEZMItJobCWWJEn1aIcBYkQMpzIFxeuArSkhM/NDBdZLGtQaNYlII+luqgkDREmSNFj1ZJqLfwdeDcygMh6wGfhDkZWSBjvnbKt/thJLkqR61JMA8dDM/Azwx8z8KvB2KnMTSuqCc7bVv65ag20lliRJg1lPsph2zEG4PiImAb8DDi6sRlKdcM62+uZUE5IkqR71JEBcFBF7AX8LLKUyLcVnCq2VJO3inGpCkiTVo24DxIjYDfh9Zj4D/Bg4ZEBqJUmDgK3EkiSp3nQ7BjEztwDnDVBdJEmSJEkl6kmSmh9ExF9HxNiI2Lvjp/CaSZIkSZIGVE/GIHbMd/iJmrLE7qaS1HCWrGp33KUkSXVshwFiZo4biIpIknZtS1a1b5O5tX39Bi686W4Ag0RJkurEDgPEiDizs/LM/Fr/V0eStKu6fNkD20zrAbBh02YuX/aAAaIkSXWiJ11MD695Pxw4DvgFYIAoSQ1k7foNvSqXJEmDT0+6mP6/tcsRMRL498JqpMI4dkhSXxwwqon2ToLBA0Y1lVAbSZJUhJ5kMd3en4Dx/V0RFatj7FD7+g0kL40dWrKqveyqSRok5s6YQNOwIduUNQ0bwtwZE0qqkSRJ6m89GYP4HSpZS6ESUE4EFhdZKfU/xw5J6quOzwp7IkiSVL96Mgbxipr3LwK/zcy2guqjgjh2SFJ/mDVtjAGhJEl1rCcB4qPA45m5ESAimiLi4Mz8TaE1U79y7JDqmeNrJUmS+kdPxiB+E9hSs7y5WqZBpPSxQ2sWw+cnwcWjKq9r7KWs/uH4WkmSpP7TkwBxaGa+0LFQfb97cVVSEWZNG8Olp01mzKgmAhgzqolLT5s8MK0saxbDd86HZx8DsvL6nfMNEtUvuhtfK0mSpN7pSRfTJyNiZmYuBYiIU4F1xVZLRSht7NBt82HTdt1bN22olE+ZPfD1UV1xfK0kSVL/6UmAeC5wbURcWV1uA84srkqqO892kdOoq3KpFxxfK0mS1H922MU0M3+dmW+gMr3F6zLzTZn5UE8OHhEnRsQDEfFQRMzrZP2nIuK+iFgTEbdFxEE16zZHxOrqz9Ka8nERcVdEPBgRN0SE3V13dSObe1cu9ULp42slSZLqyA4DxIj43xExKjOfy8w/RMReEfH3PdhvCHAVcBKV4HJOREzcbrNVQEtmTgFuBC6rWbchM6dWf2bWlP8D8PnMHA88A5y9o7qoZMddBMO2a80Z1lQpl/qo1PG1kiRJdSYys/sNIlZl5rTtyn6RmX++g/3eCFycmTOqyxcCZOalXWw/DbgyM4+qLj+XmXtut00ATwKvzswXtz9HV1paWrK1tbXb61TB1iyujDl8tq3ScnjcRY4/lCRJkgZIRKzMzJYdbdeTMYhDImKPzHy+euAmYI8e7DcGeKxmuQ04spvtzwZuqVkeHhGtwIvAgsxcAuwDrM/MF2uOaTPBYDBltgGhJEmStIvrSYD4deC2iLi6uvxB4Ks92C86Keu0uTIi3ge0AMfUFB+YmWsj4hDg9oi4G/h9L455DnAOwIEHHtiD6kqSJElSY+tJkprLgL8HXktlLOGtwEHd7lTRBoytWW4G1m6/UUS8FfgbYGZHK2X1vGurrw8DdwDTqEyvMSoiOgLbTo9Z3W9RZrZkZsvo0aN7UF1JkiRJamw7DBCrfgdsAd4JHAfc34N9VgDjq1lHdwfOAJbWblAdd/glKsHhEzXle0XEHtX3+wJHAfdlZcDkD4HTq5t+APh2D69BkiRJktSNLruYRsT/QyWomwM8BdxAJanNm3ty4GoSmfOAZcAQ4CuZeW9EzAdaM3MpcDmwJ/DNSv4ZHq1mLH0t8KWI2EIliF2QmfdVD30BcH01k+oq4Mu9vWhJkiRJ0st1mcW0Gpz9BDi7Y97DiHg4Mw8ZwPr1C7OYSpIkSWpkPc1i2l0X03dS6Vr6w4j414g4js4Tz0iSJEmS6kCXAWJm3pyZ7wYOo5Ik5q+A/SLiixFxwgDVT/1pzWL4/CS4eFTldc3ismskSZIkaRfSkyymf8zMazPzZCpZQ1cD8wqvmfrXmsXwnfPh2ceArLx+53yDREmSJElb9TSLKQCZ+XRmfikz31JUhVSQ2+bDpg3blm3aUCmXJEmSJHoZIGoQe7atd+WSJEmSGo4BYqMY2dy7ckmSJEkNxwCxURx3EQxr2rZsWFOlXJIkSZIwQGwcU2bDKQth5FggKq+nLKyUS5IkSRIwtOwKaABNmW1AKEmSJKlLtiBKkiRJkgADREmSJElSlQGiJEmSJAkwQJQkSZIkVRkgSpIkSZIAA0RJkiRJUpUBoiRJkiQJMECUJEmSJFUZIEqSJEmSAANESZIkSVKVAaIkSZIkCTBAlCRJkiRVGSBKkiRJkgADREmSJElSVaEBYkScGBEPRMRDETGvk/Wfioj7ImJNRNwWEQdVy6dGxJ0RcW913btr9rkmIh6JiNXVn6lFXoMkSZIkNYrCAsSIGAJcBZwETATmRMTE7TZbBbRk5hTgRuCyavmfgDMz83XAicAXImJUzX5zM3Nq9Wd1UdcgSZIkSY2kyBbEI4CHMvPhzHwBuB44tXaDzPxhZv6puvgzoLla/qvMfLD6fi3wBDC6wLpKkiRJUsMrMkAcAzxWs9xWLevK2cAt2xdGxBHA7sCva4o/W+16+vmI2KM/KitJkiRJja7IADE6KctON4x4H9ACXL5d+f7AvwMfzMwt1eILgcOAw4G9gQu6OOY5EdEaEa1PPvnkzl2BJEmSJDWQIgPENmBszXIzsHb7jSLircDfADMz8/ma8j8Dvgv8bWb+rKM8Mx/PiueBq6l0ZX2ZzFyUmS2Z2TJ6tL1TJUmSJGlHigwQVwDjI2JcROwOnAEsrd0gIqYBX6ISHD5RU747cDPwtcz85nb77F99DWAWcE+B1yBJkiRJDWNoUQfOzBcj4jxgGTAE+Epm3hsR84HWzFxKpUvpnsA3K/Eej2bmTGA2MB3YJyLOqh7yrGrG0msjYjSVLqyrgXOLugZJkiRJaiSR2emwwLrS0tKSra2tZVdDkiRJkkoRESszs2VH2xXZxVSSJEmSNIgYIEqSJEmSAANESZIkSVKVAaIkSZIkCTBAlCRJkiRVGSBKkiRJkgADREmSJElSlQGiJEmSJAkwQJQkSZIkVRkgSpIkSZIAA0RJkiRJUpUBoiRJkiQJMECUJEmSJFUZIEqSJEmSAANESZIkSVKVAaIkSZIkCTBAlCRJkiRVGSBKkiRJkgADREmSJElSlQGiJEmSJAkwQJQkSZIkVRkgSpIkSZIAA0RJkiRJUlWhAWJEnBgRD0TEQxExr5P1n4qI+yJiTUTcFhEH1az7QEQ8WP35QE35/4qIu6vHXBgRUeQ1SJIkSVKjKCxAjIghwFXAScBEYE5ETNxus1VAS2ZOAW4ELqvuuzfwd8CRwBHA30XEXtV9vgicA4yv/pxY1DVIkiRJUiMpsgXxCOChzHw4M18ArgdOrd0gM3+YmX+qLv4MaK6+nwH8IDOfzsxngB8AJ0bE/sCfZeadmZnA14BZBV6DJEmSJDWMIgPEMcBjNctt1bKunA3csoN9x1Tf9/SYkiRJkqQeGlrgsTsbG5idbhjxPqAFOGYH+/bmmOdQ6YrKgQceuKO6SpIkSVLDK7IFsQ0YW7PcDKzdfqOIeCvwN8DMzHx+B/u28VI31C6PCZCZizKzJTNbRo8evdMXIUmSJEmNosgAcQUwPiLGRcTuwBnA0toNImIa8CUqweETNauWASdExF7V5DQnAMsy83HgDxHxhmr20jOBbxd4DZIkSZLUMArrYpqZL0bEeVSCvSHAVzLz3oiYD7Rm5lLgcmBP4JvV2SoezcyZmfl0RPz/VIJMgPmZ+XT1/ceAa4AmKmMWb0GSJEmS1GdRSQZa31paWrK1tbXsakiSJElSKSJiZWa27Gi7IruYSpIkSZIGEQNESZIkSRJggChJkiRJqjJAlCRJkiQBBoiSJEmSpCoDREmSJEkSYIAoSZIkSaoyQJQkSZIkAQaIkiRJkqQqA0RJkiRJEmCAKEmSJEmqMkCUJEmSJAEwtOwKNKIlq9q5fNkDrF2/gQNGNTF3xgRmTRtTdrUkSZJKsWnTJtra2ti4cWPZVZEGveHDh9Pc3MywYcN2an8DxAG2ZFU7F950Nxs2bQagff0GLrzpbgCDREmS1JDa2tp45StfycEHH0xElF0dadDKTJ566ina2toYN27cTh3DLqYD7PJlD2wNDjts2LSZy5c9UFKNJEmSyrVx40b22Wcfg0OpjyKCffbZp0+t8QaIA2zt+g29KpckSWoEBodS/+jrvyUDxAF2wKimXpVLkiSpeB/60Id41atexaRJk7rcJjM5//zzOfTQQ5kyZQq/+MUvtq776le/yvjx4xk/fjxf/epXd3i+O+64g5NPPrlf6r4zrrnmGtauXVva+fvbrbfeyoQJEzj00ENZsGBBp9t87nOfY+LEiUyZMoXjjjuO3/72t1vXDRkyhKlTpzJ16lRmzpy5w/P95je/6fZZKdqSJUu47777Cjm2AeIAmztjAk3DhmxT1jRsCHNnTCipRpIkSYPLklXtHLXgdsbN+y5HLbidJava+3zMs846i1tvvbXbbW655RYefPBBHnzwQRYtWsTHPvYxAJ5++mkuueQS7rrrLn7+859zySWX8Mwzz+xUPV588cWd2q+3Sg0Q1yyGz0+Ci0dVXtcs7tPhNm/ezCc+8QluueUW7rvvPq677rpOg6dp06bR2trKmjVrOP300/n0pz+9dV1TUxOrV69m9erVLF26dKfrMlC/PwPEOjJr2hguPW0yY0Y1EcCYUU1cetpkE9RIkiT1QEfCv/b1G0heSvjX1yBx+vTp7L333t1u8+1vf5szzzyTiOANb3gD69ev5/HHH2fZsmUcf/zx7L333uy1114cf/zxnQabt956K4cddhhHH300N91009byiy++mHPOOYcTTjiBM888k40bN/LBD36QyZMnM23aNH74wx8ClaDu1FNP5cQTT2TChAlccsklW4/xuc99jkmTJjFp0iS+8IUvAC9v5briiiu4+OKLufHGG2ltbeW9730vU6dOZcOGARzqtGYxfOd8ePYxICuv3zm/T0Hiz3/+cw499FAOOeQQdt99d8444wy+/e1vv2y7N7/5zYwYMQKAN7zhDbS1tfXqPCtXruT1r389b3zjG7nqqqu2ll9zzTW8613v4pRTTuGEE04gM5k7dy6TJk1i8uTJ3HDDDUCl1Xj69Om84x3vYOLEiZx77rls2bIFgOuuu47JkyczadIkLrjggq3H3nPPPbe+v/HGGznrrLP4r//6L5YuXcrcuXOZOnUqv/71r3t1HTtiFtMSzJo2xoBQkiRpJ3SX8K/o71ft7e2MHTt263JzczPt7e1dltfauHEjH/nIR7j99ts59NBDefe7373N+pUrV7J8+XKampr4x3/8RwDuvvtufvnLX3LCCSfwq1/9CqgEQ/fccw8jRozg8MMP5+1vfzsRwdVXX81dd91FZnLkkUdyzDHHsNdee3V6HaeffjpXXnklV1xxBS0tLf1yb3rstvmwabuAdNOGSvmU2Tt1yM7u/1133dXtPl/+8pc56aSTti5v3LiRlpYWhg4dyrx585g1a9bL9vngBz/IP/3TP3HMMccwd+7cbdbdeeedrFmzhr333ptvfetbrF69mv/+7/9m3bp1HH744UyfPh2o/P7uu+8+DjroIE488URuuukm3vSmN3HBBRewcuVK9tprL0444QSWLFnSaR0A3vSmNzFz5kxOPvlkTj/99B7fp56yBVGSJEmDRpkJ/zLzZWUR0WV5rV/+8peMGzeO8ePHExG8733v22b9zJkzaWqq5KRYvnw573//+wE47LDDOOigg7YGiMcffzz77LMPTU1NnHbaaSxfvpzly5fzjne8g1e84hXsueeenHbaafzkJz/pl2vud8920WrXVXkP9OT+1/r6179Oa2vrNkHeo48+SmtrK9/4xjf45Cc/+bJWuWeffZb169dzzDHHAGz9/XToaEGGyu9vzpw5DBkyhP32249jjjmGFStWAHDEEUdwyCGHMGTIEObMmcPy5ctZsWIFxx57LKNHj2bo0KG8973v5cc//vHO3Yx+YIBYhn7udy1JktQoykz419zczGOPPbZ1ua2tjQMOOKDL8u11F7S84hWv2Pq+s4Cnq2N0FaACDB06dGsXRqBPUx/0m5HNvSvvgZ7ef4D//M//5LOf/SxLly5ljz322Fresf0hhxzCsccey6pVq7bZLzMH/Pe3/fYD9fsrNECMiBMj4oGIeCgi5nWyfnpE/CIiXoyI02vK3xwRq2t+NkbErOq6ayLikZp1U4u8hn5XQL9rSZKkRlFmwr+ZM2fyta99jczkZz/7GSNHjmT//fdnxowZfP/73+eZZ57hmWee4fvf/z4zZszYZt/DDjuMRx55ZGvL1HXXXdfleaZPn861114LwK9+9SseffRRJkyoXN8PfvADnn76aTZs2MCSJUs46qijmD59OkuWLOFPf/oTf/zjH7n55pv5i7/4C/bbbz+eeOIJnnrqKZ5//nn+4z/+Y+s5XvnKV/KHP/yhv2/Rjh13EQzbLpgf1lQp30mHH344Dz74II888ggvvPAC119/faeZSFetWsVHP/pRli5dyqte9aqt5c888wzPP/88AOvWreOnP/0pEydO3GbfUaNGMXLkSJYvXw6w9ffTmenTp3PDDTewefNmnnzySX784x9zxBFHAJUupo888ghbtmzhhhtu4Oijj+bII4/kRz/6EevWrWPz5s1cd911W1sq99tvP+6//362bNnCzTffvPUcRf7+ChuDGBFDgKuA44E2YEVELM3M2nQ7jwJnAX9du29m/hCYWj3O3sBDwPdrNpmbmTcWVfdCFdDvWpIkqVF0jDO8fNkDrF2/gQNGNTF3xoQ+jz+cM2cOd9xxB+vWraO5uZlLLrmEs88+m3/5l38B4Nxzz+Vtb3sb3/ve9zj00EMZMWIEV199NQB77703n/nMZzj88MMBuOiii16W8Gb48OEsWrSIt7/97ey7774cffTR3HPPPZ3W5eMf/zjnnnsukydPZujQoVxzzTVbW7uOPvpo3v/+9/PQQw/xnve8Z+sYwrPOOmtrEPLhD3+YadOmba3LkUceybhx4zjssMO2nuOsCTThfQAACDVJREFUs87i3HPPpampiTvvvHNr99bCdXzfvW1+pVvpyOZKcNiH78FDhw7lyiuvZMaMGWzevJkPfehDvO51rwMq19/S0sLMmTOZO3cuzz33HO9617sAOPDAA1m6dCn3338/H/3oR9ltt93YsmUL8+bNe1mACHD11VfzoQ99iBEjRrzsDwC13vGOd3DnnXfy+te/nojgsssu49WvfjW//OUveeMb38i8efO4++67tyas2W233bj00kt585vfTGbytre9jVNPPRWABQsWcPLJJzN27FgmTZrEc889B8AZZ5zBRz7yERYuXMiNN97Ia17zmp2+f9uL7po0+3TgiDcCF2fmjOryhQCZeWkn214D/EdnQV9EnAMck5nv3dG2XWlpacnW1taduYz+d/EooLN7HnDx+oGujSRJUunuv/9+Xvva15ZdjV3eNddcQ2trK1deeWXZVdFOuOOOO7jiiiu2acktSmf/piJiZWbuMCtRkV1MxwCP1Sy3Vct66wxg+zb4z0bEmoj4fETs0dlOu6wC+l1LkiRJUn8oMkDsbBRnr5orI2J/YDKwrKb4QuAw4HBgb+CCTnYlIs6JiNaIaH3yySd7c9piFdDvWpIkSfXvrLPOsvVwEDv22GMHpPWwr4oMENuAsTXLzcDaXh5jNnBzZm7qKMjMx7PieeBq4IjOdszMRZnZkpkto0eP7uVpCzRlNpyyEEaOBaLyespCxx9KkiRJKl1hSWqAFcD4iBgHtFPpKvqeXh5jDpUWw60iYv/MfDwqOV9nAZ2P7t2VTZltQChJklRjR9MISOqZvuaYKawFMTNfBM6j0j30fmBxZt4bEfMjYiZARBweEW3Au4AvRcS9HftHxMFUWiB/tN2hr42Iu4G7gX2Bvy/qGiRJklS84cOH89RTT/X5i63U6DKTp556iuHDh+/0MQrLYror2aWymEqSJGkbmzZtoq2tbdeYyF0a5IYPH05zczPDhg3bprynWUyL7GIqSZIk7dCwYcMYN25c2dWQRLFJaiRJkiRJg4gBoiRJkiQJMECUJEmSJFU1RJKaiHgS+G3Z9ejEvsC6siuhuuXzpSL5fKlIPl8qms+YirSrPl8HZeYOJ4hviABxVxURrT3JJCTtDJ8vFcnnS0Xy+VLRfMZUpMH+fNnFVJIkSZIEGCBKkiRJkqoMEMu1qOwKqK75fKlIPl8qks+XiuYzpiIN6ufLMYiSJEmSJMAWREmSJElSlQFiCSLixIh4ICIeioh5ZddH9ScifhMRd0fE6ohoLbs+Gtwi4isR8URE3FNTtndE/CAiHqy+7lVmHTV4dfF8XRwR7dXPsNUR8bYy66jBKyLGRsQPI+L+iLg3Iv6yWu5nmPqsm+drUH+G2cV0gEXEEOBXwPFAG7ACmJOZ95VaMdWViPgN0JKZu+IcPBpkImI68BzwtcycVC27DHg6MxdU/9C1V2ZeUGY9NTh18XxdDDyXmVeUWTcNfhGxP7B/Zv4iIl4JrARmAWfhZ5j6qJvnazaD+DPMFsSBdwTwUGY+nJkvANcDp5ZcJ0nqUmb+GHh6u+JTga9W33+Vyn+IUq918XxJ/SIzH8/MX1Tf/wG4HxiDn2HqB908X4OaAeLAGwM8VrPcRh08SNrlJPD9iFgZEeeUXRnVpf0y83Go/AcJvKrk+qj+nBcRa6pdUO3+pz6LiIOBacBd+Bmmfrbd8wWD+DPMAHHgRSdl9vNVfzsqM/8cOAn4RLULlyQNFl8EXgNMBR4H/rHc6miwi4g9gW8Bn8zM35ddH9WXTp6vQf0ZZoA48NqAsTXLzcDakuqiOpWZa6uvTwA3U+naLPWn/6mOvegYg/FEyfVRHcnM/8nMzZm5BfhX/AxTH0TEMCpf3q/NzJuqxX6GqV909nwN9s8wA8SBtwIYHxHjImJ34Axgacl1Uh2JiFdUB0oTEa8ATgDu6X4vqdeWAh+ovv8A8O0S66I60/HFveod+BmmnRQRAXwZuD8zP1ezys8w9VlXz9dg/wwzi2kJqqluvwAMAb6SmZ8tuUqqIxFxCJVWQ4ChwDd8xtQXEXEdcCywL/A/wN8BS4DFwIHAo8C7MtNEI+q1Lp6vY6l0zUrgN8BHO8aLSb0REUcDPwHuBrZUi/8/KuPE/AxTn3TzfM1hEH+GGSBKkiRJkgC7mEqSJEmSqgwQJUmSJEmAAaIkSZIkqcoAUZIkSZIEGCBKkiRJkqoMECVJ6qGI2BwRq2t+5vXjsQ+OiEE1V5Ykqf4MLbsCkiQNIhsyc2rZlZAkqSi2IEqS1EcR8ZuI+IeI+Hn159Bq+UERcVtErKm+Hlgt3y8ibo6I/67+vKl6qCER8a8RcW9EfD8imkq7KElSQzJAlCSp55q262L67pp1v8/MI4ArgS9Uy64EvpaZU4BrgYXV8oXAjzLz9cCfA/dWy8cDV2Xm64D1wDsLvh5JkrYRmVl2HSRJGhQi4rnM3LOT8t8Ab8nMhyNiGPC7zNwnItYB+2fmpmr545m5b0Q8CTRn5vM1xzgY+EFmjq8uXwAMy8y/L/7KJEmqsAVRkqT+kV2872qbzjxf834z5gqQJA0wA0RJkvrHu2te76y+/y/gjOr79wLLq+9vAz4GEBFDIuLPBqqSkiR1x79MSpLUc00Rsbpm+dbM7JjqYo+IuIvKH1/nVMvOB74SEXOBJ4EPVsv/ElgUEWdTaSn8GPB44bWXJGkHHIMoSVIfVccgtmTmurLrIklSX9jFVJIkSZIE2IIoSZIkSaqyBVGSJEmSBBggSpIkSZKqDBAlSZIkSYABoiRJkiSpygBRkiRJkgQYIEqSJEmSqv4vRRtXYUrQ/TIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?\n",
    "\n",
    "## Answer:\n",
    "dropout 降低了过拟合现象，具体表现在其训练准确度下降而验证准确度提升\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). If we are concerned about overfitting, how should we modify p (if at all) when we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?\n",
    "\n",
    "## Answer:\n",
    "应增大p\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
